feat: add LiteLLM messages and agents integrations - #102
andrewklatzke wants to merge 6 commits into
Conversation
Add in-process messages and Agents SDK handlers so AI Configs can route across LiteLLM-supported providers with tools, streaming, and graph support.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Register catalog metadata, document the server SDK peer, emit native graph telemetry, and capture streamed content on the root span.
Start tracing after graph setup and always end the graph span across success, failures, result processing errors, and cancellation.
mypy cannot resolve a base class read directly off a dynamically imported module, so bind the name first as the OpenAI adapter does. Co-authored-by: Cursor <cursoragent@cursor.com>
…-lite-llm # Conflicts: # .release-please-manifest.json
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3683c97. Configure here.
| model_factory=model_factory, capture_content=capture_content | ||
| ), | ||
| **kwargs, | ||
| ).invoke(user_input, context, variables) |
There was a problem hiding this comment.
Convenience wrappers drop conversation history
Medium Severity
litellm_agents and litellm_messages pop variables for invoke but leave history in kwargs for config. history is an invoke argument, so it is either rejected or never reaches the handler, and conversation history passed to the convenience APIs is lost.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 3683c97. Configure here.
| if usage_reported: | ||
| root.set_attribute("gen_ai.response.model", _model_name(config_value)) | ||
| set_usage_span_attributes(root, total) | ||
| end_span_once(root, ended, abandoned=True) |
There was a problem hiding this comment.
Stream cancel missing cancelled marker
Medium Severity
The messages stream finally treats every unfinished run as abandoned and never sets cancelled. CancelledError is not caught, so a cancelled stream is recorded as launchdarkly.stream.abandoned without launchdarkly.run.cancelled. The LiteLLM graph span has the same gap: cancel ends the span with no status and no cancelled marker.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 3683c97. Configure here.
2 participants
Summary
launchdarkly-ai-litellm-messagesusing in-process LiteLLM completionslaunchdarkly-ai-litellm-agentsusing the OpenAI Agents SDK LiteLLM modelUsage
Python runs LiteLLM in-process, so callers set the upstream provider credentials required by the evaluated model; no LiteLLM proxy is required.
Test plan
uv run pytestuv run mypyacross all package sourcesuv run ruff check .uv run ruff format --check .Made with Cursor